Image calibration

ABSTRACT

A method and apparatus for calibrating an image from a camera mounted on a vehicle using camera pose parameter determined from information relating to suspension level from a suspension system of the vehicle. The difference in the suspension level compared with a suspension level datum can be determined and an adjustment to the camera pose parameter values can be obtained from the difference in the suspension level.

TECHNICAL FIELD

This disclosure relates to a method of calibrating images, and inparticular to a method of calibrating an image from a camera mounted ona vehicle, and to a corresponding apparatus.

BACKGROUND

Driver assist camera systems are increasingly being provided in modernvehicles to provide the driver with assistance, for example in drivingor in parking the vehicle. In particular, driver assist systems are ableto provide a driver with a view of the position of the vehicle, relativeto other vehicles or to features of the road such as the curb, to helpthe driver in maneuvering or parking the vehicle. In some driver assistcamera systems, graphical data is displayed to the user, for example toindicate the projected path of the vehicle on the ground when parking orovertaking.

Driver assist camera systems typically include one or more cameras,mounted on the vehicle, that capture images of the surroundings of thevehicle. The images captured by the camera are calibrated so that theimages accurately represent the surroundings of the vehicle, and so thatthey can be combined with other images and/or with graphical data fordisplay. Driver assist camera systems typically perform a geometricalprojection of a camera image onto a plane representing the ground onwhich the vehicle is located, and may perform an overlay of additionalgraphical data, such as graphical data illustrating the projected pathof the vehicle on the ground, on the image data.

The height of the camera relative to the ground when the image was takenand the height of the camera relative to the projection plane onto whichthe image is to be projected are used to perform an accurate geometricalprojection of an image on to a plane. Inaccuracy in this heightinformation makes the alignment of composite pictures made by combiningimages from more than one camera more difficult, because the edges oroverlap between the original images will not correspond. In addition,the positioning of graphical information overlaid on a camera image oron a composite camera image can be inaccurate.

In some driver assist camera systems, the height of the camera above theground is measured in a calibration stage during manufacture. Althoughthis method is simple to carry out, the height measurement becomesinaccurate with load, ride height changes and other variations.

In other driver assist camera systems, a camera height measurement foruse in calibrating images may be updated in service from the imagestaken by the camera. One such system is disclosed in WO2009/142921,which is incorporated by reference, which relates to a system and amethod for calibrating a camera on a vehicle while the vehicle is beingdriven. Images from a camera mounted on the vehicle are analysed todetermine a ground plane, and the height of the camera above thedetermined ground plane can be found.

Although systems such as this can produce good results, the processingof the images to determine the height of the camera may take some time,and the height of the camera tends to be updated infrequently. Inaddition, techniques to analyse the position of a ground plane fromimages taken by a camera may work poorly on some surfaces. Furthermore,rotations or deflections of the camera caused by load or handlingvariations are not always considered. These factors mean that thecalibration of images from the camera may be inaccurate for periods oftime.

The present invention seeks to alleviate or ameliorate at least some ofthe disadvantages of the prior art, and to provide a method ofcalibrating images.

SUMMARY

In accordance with a first aspect of the invention, there is provided amethod comprising: calibrating an image from a camera mounted on avehicle using at least one camera pose parameter; wherein a value of theat least one camera pose parameter is determined using suspension levelinformation of a suspension system of the vehicle.

In accordance with a second aspect of the invention there is provided anapparatus comprising a camera pose parameter function, coupled toreceive suspension level information from a vehicle suspension system ofa vehicle and to determine a value of at least one camera pose parameterfrom the suspension level information; and an image processor, coupledto receive the value of at least one camera pose parameter from thecamera pose parameter function, and operable to calibrate images from acamera mounted to the vehicle using the value of the at least one camerapose parameter.

In accordance with embodiments of the invention, camera images arecalibrated using camera pose information derived from a vehicle systemto adjust images, so that images can be displayed accurately. Inaccordance with embodiments of the invention, the alignment betweencalibrated images that have been joined or combined, and/or the accuracyof registration between images and graphical data, may be improved.

The calibration of images using camera pose parameter values derivedfrom suspension level information from the suspension system may beperformed frequently and at regular intervals. The calibration thereforeis responsive to changes in the vehicle load or in the vehicle handling.Suspension data can be updated irrespective of the surface on which thevehicle stands.

In some embodiments, the values of the pose parameters of the camera areupdated by collecting information, for example via a vehiclecommunication network, from the suspension system. In some embodiments,values for the camera pose parameters are determined during productionand during in-service calibration exercises so the pose parameter valuesdetermined from both suspension and video analysis sources can becorrelated. In some embodiments, the suspension data provides highlyresponsive relative corrections to initial camera pose parameter valuesdetermined by video analysis based calibration.

The implementation of an embodiment of the invention in a vehicle thatalready has suspension data available within a vehicle communicationsnetwork involves minimal marginal cost per vehicle since no extraphysical elements are required.

Embodiments of the disclosure are described below, with reference to theaccompanying drawings, by way of example only.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a side elevation view of a camera system.

FIG. 2 is a schematic diagram illustrating the processing of imagescaptured in the camera system of FIG. 1.

FIG. 3 is a side elevation view of a camera system illustrating a changein ground plane.

FIG. 4 is a schematic diagram illustrating the processing of imagescaptured in the camera system of FIG. 3.

FIG. 5 is a side elevation view of a camera system illustrating avariation in the suspension level of a vehicle suspension system.

FIG. 6a is a schematic diagram illustrating a vehicle datum plane of thevehicle.

FIG. 6b is a schematic diagram showing the effect of change in thevehicle pitch on the vehicle plane.

FIG. 6c is a schematic diagram showing the effect of change in thevehicle roll on the vehicle plane.

FIG. 7 is a schematic block diagram of an apparatus that can be used toimplement the present invention, in accordance with one embodiment ofthe invention.

FIG. 8 is a flow chart illustrating the steps of a method in accordancewith one embodiment of the invention.

DETAILED DESCRIPTION

In embodiments of the invention, images from a camera are calibratedusing camera pose parameter values derived from suspension levelinformation. The calibrated images can be accurately combined with atleast one other image and/or with graphical data.

The combination of position and orientation of an object is referred toas the pose of the object. Thus the camera pose describes the positionand orientation of the camera in 3-dimensional (3D) space. Variations inthe position and orientation of the camera in 3D space can be describedusing three position pose parameters, describing the position of thecamera along the x, y, and z axes respectively, and three orientationpose parameters, describing the pitch, roll, and yaw of the camerarespectively. In the context of the calibration of a camera, the mostimportant parameters are the camera height pose parameter, and the pitchand roll pose parameters.

The principles involved in combining images from different cameras willbe explored in more detail with reference to FIGS. 1 and 2. In thisexample, only a single camera pose parameter, relating to the cameraheight, is considered.

FIG. 1 is a side elevation view of a camera system having a first camera1 and second camera 2.

The first camera 1 is mounted at a distance H₁ from a datum Z, with itsoptical axis 3 being perpendicular to the datum Z. The first camera 1has a field of view shown by dashed lines 4 a, 4 b. The second camera 2is mounted at a distance H₂ from the datum Z, with its optical axis 5 atan angle to the datum Z. The second camera 2 has a field of view shownby dashed lines 6 a, 6 b. The distance separating the cameras 1, 2 inthe direction parallel to the ground plane G is δI.

A ground plane G is provided, which represents the ground level on whichthe vehicle sits. In this illustration, the ground plane G is the datumZ.

The cameras 1, 2 both take images of a part of the ground plane Gcontaining a circular object P. Dashed line 7 represents the path oflight from point P to camera 1 and dashed line 8 represents the path oflight from point P to camera 2. The angle formed between path of light 7and optical axis 3 is shown as θ1, and the angle formed between path oflight 8 and optical axis 5 is shown as θ2.

A geometrical transformation of an image captured by camera 1 or camera2 will now be explained in more detail. The illustrated geometricaltransformation is a plane projection i.e. images from the cameras 1, 2are geometrically transformed so as to be displayed as if on aprojection plane V, as shown in FIG. 1, viewed from a point overhead. Inthis illustration the projection plane V used for the overhead view isthe same as the ground plane G as shown in FIG. 1. It is not necessarythat the projection plane V is the ground plane G, but this is anespecially useful case for road vehicles. However, the principlesdescribed are also applicable to projections onto other planes.

FIG. 2 illustrates the processing of the images captured by cameras 1, 2to produce a merged image.

Images 10 and 11 represent the images captured by respective cameras 1,2 of the arrangement shown in FIG. 1. In image 10 point P is naturallyforeshortened owing to the angle of view of the camera 1, and is shownas an oval point Pa 12. In image 11 point P is naturally foreshortenedowing to the angle of view of the camera 2, and is shown as an ovalpoint Pb 13.

Images 14 and 15 represent the projection of the respective image 10, 11on the projection plane V as if viewed from overhead, which removes theforeshortening effect caused by the angle of view of the cameras.

Techniques useful in carrying out a projection of an image onto a planewill be known to a skilled person. For example, in some embodiments apixel-by-pixel mapping may be used to carry out a projection onto theprojection plane V, as will be described in more detail with referenceto FIGS. 1 and 2.

Thus, to identify picture information for a point within an image in theprojection plane V, firstly the angle between a ray of light travellingfrom the point on the recorded image to the camera and the optical axisof the camera recording the image is determined. The calculation of thisangle depends upon the height of the camera above the ground plane G,since the point is in the ground plane G. The height of the camera abovethe ground plane is the same as the height of the camera above the datumZ in FIG. 1, and so the angles can be calculated accurately.

Point P is represented as point Pa 12 in image 10 in FIG. 2, and theangle formed between path of light 7 and optical axis 3 of camera 1 isangle θ1, as described above with reference to FIG. 1. Point P isrepresented as point Pb 13 in image 11 in FIG. 2, and the angle formedbetween path of light 8 and optical axis 4 of camera 2 is shown as angleθ2 as described above with reference to FIG. 1.

Next, using the angle between a ray of light travelling from the pointon the recorded image to the camera and the optical axis of the camerarecording the image, together with a knowledge of the distance of theprojection plane V away from the cameras 1, 2, the picture informationin recorded images 10 and 11 can be transformed into picture informationin projected images 14, 15 respectively. It can be seen in FIG. 2 thatpoint P is transformed from an oval point Pa 12 in recorded image 10 toa circular point Pc 16 in projected image 14, and that point P istransformed from an oval point Pb 13 in recorded image 11 to a circularpoint Pd 17 in projected image 15.

As will be apparent, where the projection plane V is the same as theground plane G, the distribution of points within the projection plane Vwill be proportional to the corresponding positions on the ground planeG.

The spatial relationship between these projected images 14, 15 is knownfrom the distance δI separating the cameras 1, 2 in the directionparallel to the ground plane G and from the angle (not shown) at whichthe cameras 1, 2 are oriented with respect to each other.

Taking the spatial relationship between the projected images 14, 15 intoaccount, the projected images 14, 15 may be combined to form a combinedor merged output image 18. Where the projected images 14, 15 overlap, itis possible to join them together using a number of differenttechniques, as will be apparent to a skilled person. In someembodiments, in order to achieve a good alignment of the projectedimages 14, 15, the picture information from the projected images 14, 15in overlapping regions can be combined using alpha blending, or othertechniques for combining picture information from two images.

FIG. 2 shows the merged image 18 formed by combining the projectedimages 14 and 15 as described above. The projected point Pc 16 inprojected image 14 and projected point Pd 17 in projected image 15overlap and therefore are seen as a single point P 19 in the mergedimage 18.

The above described system works well provided that the measurements onwhich the projection onto the plane V is based are accurate. Problemsthat may arise in combining images that have been projected onto aprojection plane will now be described with reference to FIGS. 3 and 4,which correspond to FIGS. 1 and 2 described previously. These problemsarise owing to variability in the height above the projection plane,such as occurs if the cameras are mounted on a vehicle. As will beunderstood by a skilled person, similar problems arise owing tovariability in the other camera pose parameter values, as will bedescribed further with reference to FIG. 6.

As will be understood by a skilled person, a vehicle body is mounted onsprings and tyres so at times the ground plane G shown in FIG. 1, whichrepresents the actual ground level on which a vehicle is located, is nolonger the same as the datum Z shown in FIG. 1, which represents the“nominal” ground level on which the vehicle is located. As a result therespective measurements of the height of the cameras 1, 2 relative tothe datum Z are no longer an accurate measure of the distance betweenthe cameras 1, 2 and the ground. Instead the actual height between thecameras mounted on the vehicle and the ground will vary with time.

In FIG. 3, the point P is on the ground plane G that is now closer tothe cameras 1, 2 than the datum Z. It will be understood that FIG. 3shows a situation where a vehicle body, and therefore also the cameras1, 2 mounted to the vehicle body, is closer to the ground than expected,for example because of a load. As mentioned above, for simplicity, FIG.3 illustrates a vertical displacement of the cameras 1, 2, although anangular deflection of the cameras 1, 2, which might occur when a vehicleon which the cameras 1, 2 are mounted has tilted, causes similarproblems. In practice, vertical displacements and angular deflectionsand the resulting variation in camera pose parameter values, both occuras a result of loads or variations in the driving or handling of thevehicle.

As shown in FIG. 3, the change in the position of the ground plane Grelative to the datum Z results in a change in the angle between theincident ray of light from point P at the cameras 1, 2 and therespective optical axis 3 of camera 1 or the optical axis 5 of camera 2.The angular difference between the situation where the ground plane Gcontaining point P is at the datum Z as shown in FIG. 1, and thesituation where the ground plane G containing point P is at a differentheight from the datum Z, as shown in FIG. 3, is labelled as angle εθ1for camera 1 and is labelled as angle εθ2 for camera 2 respectively.

FIG. 4 illustrates the processing of images captured from cameras 1, 2to produce a merged image.

Images 20 and 21 represent the images captured by respective cameras 1,2 of the arrangement shown in FIG. 3. In image 20, point P is naturallyforeshortened owing to the angle of view of the camera 1, and is shownas an oval point Pe 22. In image 21 point P is naturally foreshortenedowing to the angle of view of the camera 2, and is shown as an ovalpoint Pf 23. The position of points Pa 12, Pb 13 as shown in FIG. 2,which represent the position of point P in the image captured by cameras1, 2 when the ground plane is on the datum Z are also shown in FIG. 4for reference.

As described above with reference to FIG. 2, techniques useful forcarrying out a projection to a plane will be known to a skilled person.In one technique as described above, in order to identify pictureinformation for a point within an image in the projection plane V,firstly the angle between a ray of light travelling from the point onthe recorded image to the camera and the optical axis of the camerarecording the image is determined.

In the situation shown in FIG. 3, the angle between the respectiveoptical axis of cameras 1, 2 and incident light from the point P on theground plane G has changed by the angle εθ1 for camera 1 and by theangle εθ2 for camera 2, respectively.

When the angle between incident light from the point P to the camera andthe optical axis of the camera recording the image is used with thedistance of the projection plane V away from the camera, the pictureinformation in recorded images 20 and 21 can be transformed into pictureinformation in projected images 24, 25 respectively.

Images 24 and 25 represent the projection of the respective image 20, 21onto the projection plane V as if viewed from overhead, which removesthe foreshortening effect caused by the angle of view of the cameras 1,2. However, in the situation shown in FIGS. 3 and 4, the projectionplane V is at the same height as the datum Z, and is therefore not atthe same height relative to the cameras 1, 2 as the ground plane G.

Thus, in FIG. 4 point P is transformed from an oval point Pe 22 inrecorded image 20 to a circular point Pg 26 in projected image 24, andpoint P is transformed from an oval point Pf 23 in recorded image 21 toa circular point Ph 27 in projected image 25. The position of points Pc16, Pd 17, which represent the correct position of point P in theprojected images 24, 25, are also shown in FIG. 4 for reference. It canbe seen that the change in angle of the incident light at the cameras 1,2 caused by the change in the height of the ground plane G relative tothe cameras 1, 2 has resulted in the projected point Pg 26 in projectedimage 24 not being in the same position as point Pc 16 in projectedimage 14, and the projected point Ph 27 in projected image 25 not beingin the same position as point Pd 17 in projected image 15.

Thereafter, taking the spatial relationship between the projected images24, 25 into account, the projected images 24, 25 may be combined to forma combined or merged output image 28, in a similar manner to thatdescribed above with reference to the combining of projected images 14,15 to form the merged image 18.

FIG. 4 shows the merged picture 28 formed by combining the projectedimages 24 and 25 as described above. As is clearly seen from the mergedpicture 28, when the projected images 24, 25 are merged, not only hasthe image of point P moved from its correct position, but it is now adouble image, as the projected images Pg 26, Ph 27 of point P are indifferent places.

The problem of the misaligned projection can be corrected if the camerapose parameter values are accurate and up to date. As remarked earlier,for rigid vehicles, the only measurements that change quickly orsubstantially are those affected by vehicle body movements, and the mainelement of that is the compression of the vehicle suspension.

Suspension level information can be used to correct the pose parametervalues for cameras 1, 2 in the calculations, or the suspension levelinformation can be used to move the projection plane V to the positionof the ground plane G.

FIG. 5 shows how errors in camera calibration can be overcome bycorrecting camera pose parameter values using the variation in thesuspension level of a vehicle suspension. FIG. 5 relates to the cameraheight parameter as an exemplary camera pose parameter. The arrangementshown in FIG. 5 generally corresponds with the arrangement shown in FIG.3, and therefore features in common with FIG. 3 will not be described indetail with reference to FIG. 5.

In FIG. 5 a suspension datum S is shown, which bears a fixedrelationship with the position of cameras 1 and 2. A single wheel of thevehicle to which the cameras 1 and 2 are mounted is shown in twodifferent positions, namely a first position 30 and a second position32, illustrating the change in the vertical height of the wheel of thevehicle relative to the suspension datum over time in response tochanges in the loading of the vehicle or in the vehicle handling, forexample.

In a first position 30, the wheel is in a position where it rests on aground level corresponding to the datum Z. At this position, theoperating height of the suspension Hs is known or can be measured. InFIG. 5 the operating height of the suspension Hs is shown as the heightdifference between the centre of the wheel in first position 30 and thesuspension datum S.

In a second position 32, the position of the ground plane G has beenraised relative to the datum Z, corresponding to a situation where avehicle body, and therefore also the cameras 1, 2 mounted to the vehiclebody, is closer to the ground than expected, for example because of aload. In this position, the centre of the wheel is closer to thesuspension datum S by a distance of εHs, as is shown in FIG. 5, relativeto the initial operating height of the suspension Hs when the wheel isat position 30.

Since the position of the suspension datum S is fixed relative to thecameras 1, 2, the change in the operating height of the suspensionbetween positions 30 and position 32 can be taken as a measure of thecamera deflection relative to the ground plane G. As a result thedifference εHs in the operating height of the suspension Hs can be usedto adjust the values of camera pose parameters, and/or to adjust theposition of the projection plane V.

Therefore, in embodiments of the invention, any variation in the camerapose parameter values may be corrected by direct measurement of theoperating height of the suspension system of the vehicle. The directmeasurements may be provided by existing systems on the vehicle, andpassed via electronic network to the picture processing system.

In some embodiments, measurement of an initial baseline position of bothcamera positions H1 and H2, and operating height of the suspensionsystem Hs is necessary and these measurements may be stored in, or beaccessible for use by, a picture processing system.

In some embodiments, an initial camera pose measurement may be carriedout as an initial measurement when the vehicle is built. In someembodiments, an initial camera pose measurement may be taken duringoperation, for example the camera pose may be measured by analysingimages captured by the camera. In some embodiments, the vehicle designinformation may be accurate enough to avoid the need to make anyphysical measurement to determine an initial camera pose.

During operation, the current suspension operating level may bemeasured. By comparison with the datum suspension level, the differencebetween the current suspension operating level and a previous suspensionoperating level can be determined. In these embodiments, the absolutevalue of suspension operating level Hs is not important as only thedifference or change εHs in the suspension operating level is required.

FIG. 5 shows only one wheel and suspension operating level measurementHs for the sake of clarity. In some embodiments individual measurementsfor some or all parts of the suspension can be taken allowing verticaland angular deflection at each camera position to be determined.

FIGS. 6a, 6b, and 6c are schematic diagrams illustrating the effect ofload and handling variations on the values of camera pose parameters.

FIG. 6a is a schematic diagram illustrating the ground plane G and aboveit the vehicle datum plane 34 a of the vehicle at its datum position.The wheel suspension heights 36 a, 36 b, 36 c, and 36 b are shown asarrows and each wheel suspension height is at its respective referenceposition. As discussed, this can be the suspension height of therespective wheels during a factory calibration.

Four cameras 38 a, 38 b, 38 c, and 38 d are shown, which are rigidlyattached to the vehicle at various positions. The cameras 38 a, 38 b, 38c, and 38 d have known X, Y, Z position parameter values, and Pitch,Roll, Yaw orientation parameter values relative to the vehicle datumplane 34 a. Together, the position parameters and the orientationparameters form the pose parameters of the respective camera.

FIG. 6b is a schematic diagram showing the effect of change in thevehicle pitch on the vehicle plane 34 b, with the original datum plane34 a shown as a dotted line. The change in the vehicle pitch couldresult from loading the boot of the vehicle, for example. The wheelsuspension heights 36 a, 36 b, 36 c, and 36 b have changed, causing thevehicle plane 34 b to change pitch angle with respect to the datumvehicle plane 34 a. The cameras have therefore changed height and pitchangle, since they are rigidly connected to the plane of the vehicle.

The position of the new vehicle plane 34 b can be computed from thewheel suspension heights 36 a, 36 b, 36 c, and 36 b using geometricrelationships. The new values for the camera pose parameters for each ofthe cameras 38 a, 38 b, 38 c, and 38 d can be computed from the newvehicle plane 34 b. The corrected values for the camera pose parameterscan be used instead of the datum camera pose parameter values to providea more accurate mapping for a merged surround view.

FIG. 6c is a schematic diagram showing the effect of change in thevehicle roll on the plane of the vehicle. This change in the vehicleroll could result from a driver or passenger sitting in the vehicle, forexample. The wheel suspension heights 36 a, 36 b, 36 c, and 36 b havechanged, causing the vehicle plane 34 c to change roll angle withrespect to the datum vehicle plane 34 a. The cameras have thereforechanged height and roll angle since they are rigidly connected to theplane of the vehicle.

The position of the new vehicle plane 34 c can be computed from thewheel suspension heights 36 a, 36 b, 36 c, and 36 b using geometricrelationships. The new values for the camera pose parameters for each ofthe cameras 38 a, 38 b, 38 c, and 38 d can be computed from the newvehicle plane 34 c. The corrected values for the camera pose parameterscan be used instead of the datum camera pose parameter values to providea more accurate mapping for a merged surround view.

The suspension heights 36 a, 36 b, 36 c, and 36 b of the 4 wheels can beused to compute the deflected plane of the vehicle with any combinationof pitch and roll, as will be apparent to a skilled person, and so acomposite angular deflection of the vehicle plane, illustrating acombination of pitch and roll, has not been shown.

As will be apparent to a skilled person, the camera pose parametersrelating to the height, the pitch, and the roll of the camera are thecamera pose parameters that are most affected by variations in thevehicle load or handling.

In some embodiments, as described above, the suspension height of eachof the four wheels may be measured separately, in order to allowvertical and angular deflection at each camera position to bedetermined. In other embodiments only suspension heights at wheels thatare most affected by load are measured. Typically the rear wheels aremore affected by load than the front wheels. In other embodiments, forexample on multi-wheeled vehicles, the suspension height of one wheel ofa co-located set, or of a bogie, may be measured. In embodimentsrelating to multi-suspension vehicles, for example a forward controltruck with suspended cab, the height of both the wheel suspension andthe cab suspension may be determined independently.

In some cases, for example sophisticated limousines, the suspensionheight may not be a measurement of the actual position, but a height ormode value set by the driver or an automatic system, and the suspensionsystem itself is then relied upon to maintain this height.

In some embodiments, the suspension information may be used to derivevehicle load information, and then vehicle load information and thecurrent or intended tyre pressure can be used to estimate the change inpose parameter values from compression of the tyres, to improve theaccuracy of the overall measurement. Some vehicles have tyre pressuresensors that could be useful for this.

An embodiment of the invention will now be described with reference toFIGS. 7 and 8.

FIG. 7 is a schematic block diagram of an apparatus that can be used toimplement the present invention, in accordance with one embodiment ofthe invention.

An image calibration apparatus 40 is provided for calibrating imagesfrom a camera mounted on a vehicle, as will described in more detail inthe following description. Typically the image calibration apparatus 40is mounted on the vehicle, and may be implemented either as astand-alone system or as a function of another apparatus, as selected bya skilled person.

In this embodiment, the image calibration apparatus 40 is coupled to afirst camera 41 mounted on a vehicle and to a second camera 42 mountedon the vehicle, and is arranged to receive images captured by cameras 41and 42. Clearly, the image calibration apparatus 40 may be coupled toany number of cameras in different embodiments.

The image calibration apparatus 40 is also coupled to a vehicle system43, and is arranged to receive suspension level information 44 from thevehicle system 43. In one embodiment, the vehicle system 43 is thevehicle suspension system. However, any vehicle system that is able toprovide suspension level information suitable for use in an embodimentof the present invention may be used. In particular, the vehicle system43 may in some embodiments be a vehicle dynamics system.

Generally, it is likely that the suspension level information 44 isinformation that the vehicle system 43 is already gathering orproducing. In some embodiments, it may be necessary to process data fromvehicle system 43 to obtain suspension level information 44. Thisprocessing may be carried out by vehicle system 43 or by imagecalibration apparatus 40, as appropriate. The suspension levelinformation 44, in some embodiments, may be information relating to theoperating height of the vehicle suspension.

The image calibration apparatus 40 is also coupled to an image display45. The image display 45 is arranged to display to a user, typically thedriver of the vehicle, the calibrated images, or a combination ofcalibrated images with other calibrated images and/or graphical data,which are generated by the image calibration apparatus 40.

The image calibration apparatus 40 is provided with an image store 46for storing the image data used in and produced by the image calibrationprocess carried out by the image calibration apparatus 40. Thus theimage store 46 is provided with a captured image store 47 arranged toreceive and store captured images from the first camera 41 and thesecond camera 42. The image store 46 is also provided with a projectedimage store 48 arranged to store projected images created during theimage calibration process carried out by the image calibration apparatus40, as will be discussed below in more detail. The image store 46 isalso provided with a combined image store 49 arranged to store combinedimages created during the image calibration process carried out by theimage calibration apparatus 40, as will be understood by a skilledperson. The combined image store is coupled to the image display 45 tosupply combined images for display to a user.

In some embodiments, the image store 46 is also provided with agraphical data store 51 for storing graphical data generated bygraphical data generator 52. The graphical data generator 52 is shown aspart of the image calibration apparatus 40 in the exemplary embodiment,but in some embodiments the graphical data generator 52 may be providedexternally to the image calibration apparatus 40, as will be apparent toa skilled person.

The image store 46 may be implemented in many different forms. In oneembodiment, the image store 46 may be implemented as an image array. Insome embodiments, the projected image store 48 and the combined imagestore 49 may be combined. In such a situation, for example which mightoccur when combining projected image from cameras as described withreference to FIGS. 2 and 4, projected image data from a first image maybe written to positions in the image array corresponding to the outputimage positions. Projected image data from a second captured image thatis to be combined with the projected image data from the first capturedimage is then written to the appropriate positions in the image arraycorresponding to the output image positions. Where projected imagesoverlap, output data for a pixel in the output image can be formed froma combination of the projected image data, as will be known by a skilledperson.

The image calibration apparatus 40 is also provided with an imageprocessor 53, which carries out the processing of the images asdescribed. In the illustrated embodiment, the image processor 53 isprovided with an image projector 54, which is coupled to the capturedimage store 47 and to the projected image store 48. The image projector54 is arranged to access captured images stored in the captured imagestore 47, and forms projected images by projecting the captured imagesonto a ground plane using camera pose parameter values for therespective camera that captured the image. The image projector 54 storesthe projected images in the projected images store 48.

In the illustrated embodiment, the image processor 53 is also providedwith an image combiner 55, which is coupled to the projected image store48 and the graphical data store 51 as well as to the combined imagestore 49. The image combiner 55 is arranged to access one or moreprojected images in the projected images store 48 and to combine theprojected images using camera pose parameter values to form combinedimages. The image combiner 53 is also arranged to access graphical datain the graphical data store 51, where available, and to combine thegraphical data with one or more projected images from the projectedimages store 48, using camera pose parameter values to form outputcombined images. The image combiner 53 stores the combined images in thecombined images store 49.

As discussed above, some or all elements of the function of imagecombiner 53 may be carried out by image projector 54. It will beunderstood that the projection and combination functions can beimplemented in image processor 53 in a variety of ways within the scopeof the invention.

The image calibration apparatus 40 is also provided with a camera poseparameter function 56, for determining values for the camera poseparameters 57 used in the image projection carried out by imageprojector 54 of image processor 53, and in the image combination carriedout by the image combiner 55 of image processor 53. The camera poseparameter function 56 is coupled to the vehicle system 43 to receivesuspension level information from the vehicle system 43. Typically, eachcamera mounted on the vehicle will have stored values for a set ofcamera pose parameters. The camera pose parameter function 56 is alsocoupled to the image processor 53 to supply camera pose parameter values57 to the image processor 53.

As described above with reference to FIGS. 6a, 6b, and 6c , in someembodiments of the invention, current suspension level information canbe used to determine the current position and orientation of the vehicleplane, and therefore of the cameras. The position and orientation of thevehicle plane can be compared with the position and orientation of thedatum vehicle plane to compute an offset to the values of the camerapose parameters to determine current values for the camera poseparameters.

In some embodiments not all camera pose parameter values are updatedusing suspension level information.

The functions of the image calibration apparatus 40 can be implementedin many ways, as will be apparent to a skilled person. In particular,the functions of the image calibration apparatus 40 can be implementedas a stand-alone apparatus or as a distributed system. The camera poseparameter function 56 can be implemented as a function in anothervehicle system.

FIG. 8 is a flow chart illustrating the steps of a method in accordancewith one embodiment of the invention.

As will be clear from the preceding description, camera images 60captured by a camera mounted on a vehicle are calibrated in step s61using camera pose parameter values 62 that are determined fromsuspension level information, typically from a suspension system of thevehicle. The resulting calibrated images 63 may be combined in step s64,using the camera pose parameter values 62, to form combined images 65.In some embodiments, the calibrated images may be combined withgraphical data 66 in step s64, using the camera pose parameter values62, to form calibrated combined images 65.

The calibration of an image from a camera using pose parameter valueswill be known to a skilled person. As a result, the calibration of animage from a camera using pose parameter values 62 will not be describedfurther in detail.

In embodiments of the invention, the camera pose parameter values 62used in the step of calibrating s61 and in the step of combining s64 aredetermined using suspension level information from a suspension systemof the vehicle. Typically, the camera pose parameter values 62 areperiodically adjusted to take into account variations in the pose of thecamera caused by load or handling variations, which otherwise wouldcause errors in the calibrated camera images. This adjustment usesinformation relating to the variations in the suspension level of thevehicle. In particular, in some embodiments a difference in thesuspension level compared with a suspension level datum is determinedand camera pose parameter values are updated using the suspension leveldifferences.

In one embodiment, a camera height parameter may be obtained by applyinga determined difference in the suspension level to a camera heightparameter datum at a known height relative to the ground. In thisembodiment, variations in the suspension level of the vehicle, which caneasily be measured, are used to determine variations in the height ofthe camera, which cannot easily be measured.

In order to determine the relationship between the suspension level andthe values of pose parameters of the camera, a suspension level datum isdetermined in step s67 and a corresponding camera pose datum isdetermined in step s68. Typically, the suspension level datum and thecamera pose datum are determined simultaneously, as shown by the dottedlines joining step s67 and step s68.

The step of determining the suspension level datum typically involvesmeasuring the operating height of at least one wheel of the suspensionsystem relative to the ground.

The values of the camera pose parameters when the camera is at thecamera pose datum may be measured. These camera pose parameter valuescan be stored as initial camera pose parameter values, as shown by thedotted line between step s68 and the camera pose parameter values 62.Subsequent changes in the suspension level relative to the suspensionlevel datum can be used to determine changes in the camera poseparameter values relative to the camera pose datum to obtain the currentvalues for the camera pose parameters.

The suspension level datum is the suspension level when the camera is atthe camera pose datum with known values for the camera pose parameters.Typically, the suspension level datum has a known relationship relativeto the camera pose datum.

Typically the determination of the suspension datum and the camera posedatum can be carried out at the same time, for example duringmanufacturing of the vehicle, or at the time the image calibrationapparatus 40 is fitted to the vehicle.

In some embodiments the offset from or difference between the currentsuspension level compared with the suspension level datum for each wheelis tracked and determined. In other embodiments, such as the one shownin FIG. 8 the current suspension level, and only the difference betweensuccessive suspension levels for each wheel is calculated.

Therefore in the embodiment shown in FIG. 7 when the values of camerapose parameters 62 are to be updated, in step s70 a new suspension levelis obtained.

Next in step s71 the new suspension level is compared with the currentsuspension level 72 to determine the difference in the suspension level.Once the difference in the suspension level has been determined in steps71, the new suspension level obtained in step s70 can be stored as thesuspension level, ready for the next iteration of adjusting the recordedvalues of the camera pose parameters.

The suspension level difference can then be used in a camera poseparameter adjustment step s74 to adjust the values of the camera poseparameters.

In a first camera pose parameter adjustment step s75, the current valuesof the camera pose parameters are read.

In a second camera pose parameter adjustment step s76, the currentvalues of the camera pose parameter are updated using the determinedsuspension level difference 73.

In some embodiments, this step is achieved by calculating a new plane ofthe vehicle based on the suspension level measurements, and determiningthe change in the vehicle plane. From this, the camera pose parameteradjustment can be determined for each of the camera pose parameters.

Thereafter the determined camera pose parameter adjustment can beapplied to the current value of the camera pose parameter to obtain anupdated value for the or each camera pose parameter.

If the camera pose parameter adjustment is a positive number, the valueof camera pose parameters are updated by adding the camera poseparameter adjustment to the existing value of the respective camera poseparameter. If the camera pose parameter adjustment is a negative number,the value of the camera pose parameter is updated by subtracting thecamera pose parameter adjustment from the existing value of therespective camera pose parameter.

In a third camera height adjustment step s77, the updated values for thecamera pose parameters can be stored as the current camera poseparameter values 62 to be used in calibrating images in step s61.

Thus a new method and apparatus for calibrating an image from a cameramounted on a vehicle using camera pose parameters is disclosed in whicha value of the at least one camera pose parameter is determined usingsuspension level information of a suspension system of the vehicle andan image from a camera mounted on a vehicle is calibrated using the atleast one camera pose parameter.

Other variations and modifications will be apparent to the skilledperson. Such variations and modifications may involve equivalent andother features that are already known and which may be used instead of,or in addition to, features described herein. Features that aredescribed in the context of separate embodiments may be provided incombination in a single embodiment. Conversely, features that aredescribed in the context of a single embodiment may also be providedseparately or in any suitable sub-combination.

It should be noted that the term “comprising” does not exclude otherelements or steps, the term “a” or “an” does not exclude a plurality, asingle feature may fulfil the functions of several features recited inthe claims and reference signs in the claims shall not be construed aslimiting the scope of the claims. It should also be noted that theFigures are not necessarily to scale; emphasis instead generally beingplaced upon illustrating the principles of the present invention.

The invention claimed is:
 1. A method comprising: receiving suspension level information of a suspension system of a vehicle, the suspension level information comprising an operating height of at least one wheel of the suspension system relative to a vehicle datum plane, the vehicle datum plane being a vehicle level when a plurality of cameras of the vehicle are at respective known positions and orientations relative to the ground, the vehicle datum plane having a fixed relationship with the position of the plurality of cameras; determining a new vehicle plane based on the received suspension level information; determining at least one camera pose parameter of the plurality of cameras mounted on the vehicle based on a difference between the new vehicle plane and the vehicle datum plane; and calibrating an image captured from at least one of the plurality of cameras mounted on a vehicle using the at least one camera pose parameter.
 2. The method as claimed in claim 1, comprising determining a suspension level difference between a current suspension level and a previous suspension level of the vehicle; for at least one camera pose parameter, determining a camera pose parameter adjustment from the determined suspension level difference; and determining an updated value of the at least one camera pose parameter by applying the determined camera pose parameter adjustment to a current value of the at least one camera pose parameter.
 3. The method as claimed in claim 2, wherein determining the updated value of the at least one camera pose parameter comprises: reading the current value of the camera pose parameter; forming an updated value of the at least one camera pose parameter by applying the determined camera pose parameter adjustment to the current value of the at least one camera pose parameter; and storing the updated value of the at least one camera pose parameter as a new current value of the camera pose parameter.
 4. The method as claimed in claim 1, further comprising an initial step of determining a value of the at least one camera pose parameter when the camera is at the camera pose datum.
 5. The method as claimed in claim 1, wherein the vehicle datum plane is at a known height difference relative to a camera height parameter datum.
 6. The method as claimed in claim 1, wherein the value of the at least one camera pose parameter used to calibrate the image is the value of the at least one camera pose parameter associated with the suspension level at the time the image is captured.
 7. The method as claimed in claim 1, wherein calibrating the image from the camera comprises: projecting the image on a plane using the value of the at least one camera pose parameter determined using suspension level information from a suspension system of the vehicle.
 8. The method as claimed in claim 1, further comprising combining calibrated images with other image data using the value of the at least one camera pose parameter determined using suspension level information from the suspension system of the vehicle.
 9. The method as claimed in claim 8, wherein the other image data is graphical data.
 10. The method as claimed in claim 8, wherein the other image data is a calibrated image from another camera mounted on the vehicle.
 11. The method as claimed in claim 1, further comprising displaying images to a user.
 12. An apparatus comprising: a camera pose parameter function, coupled to receive suspension level information from a vehicle suspension system of a vehicle, the suspension level information comprising an operating height of at least one wheel of the suspension system relative to a vehicle datum plane, the vehicle datum plane being a vehicle level when a plurality of cameras of the vehicle are at respective known positions and orientations relative to the ground, the vehicle datum plane having a fixed relationship with the position of the plurality of cameras, the camera pose function further configured to determine a new vehicle plane based on the received suspension level information, and to determine a value of at least one camera pose parameter of the plurality of cameras based on a difference between the new vehicle plane and the vehicle datum plane; and an image processor, coupled to receive the value of the at least one camera pose parameter from the camera pose parameter function, and operable to calibrate images captured from at least one of the plurality of cameras mounted to the vehicle using the value of the at least one camera pose parameter. 